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9 Sums and Asymptotics 

Sums and products arise regularly in the analysis of algorithms, financial applica- 
tions, physical problems, and probabilistic systems. For example, we have already 
encountered the sum 1+2 + 4 + -- - + 7V when counting the number of nodes 
in a complete binary tree with N inputs. Although such a sum can be represented 
compactly using the sigma notation 

log AT 
;=0 

it is a lot easier and more helpful to express the sum by its closed form value 

2N - 1. 

By closed form, we mean an expression that does not make use of summation 
or product symbols or otherwise need those handy (but sometimes troublesome) 

dots Expressions in closed form are usually easier to evaluate (it doesn't get 

much simpler than 2N — 1, for example) and it is usually easier to get a feel for 
their magnitude than expressions involving large sums and products. 

But how do you find a closed form for a sum or product? Well, it's part math and 
part art. And it is the subject of this chapter. 

We will start the chapter with a motivating example involving annuities. Figuring 
out the value of the annuity will involve a large and nasty-looking sum. We will then 
describe several methods for finding closed forms for all sorts of sums, including 
the annuity sums. In some cases, a closed form for a sum may not exist and so we 
will provide a general method for finding good upper and lower bounds on the sum 
(which are closed form, of course). 

The methods we develop for sums will also work for products since you can 
convert any product into a sum by taking a logarithm of the product. As an example, 
we will use this approach to find a good closed-form approximation to 

n\ ::= 1 • 2 • 3 • • 

We conclude the chapter with a discussion of asymptotic notation. Asymptotic 
notation is often used to bound the error terms when there is no exact closed form 
expression for a sum or product. It also provides a convenient way to express the 
growth rate or order of magnitude of a sum or product. 
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9.1 The Value of an Annuity 

Would you prefer a million dollars today or $50,000 a year for the rest of your life? 
On the one hand, instant gratification is nice. On the other hand, the total dollars 
received at $50K per year is much larger if you live long enough. 

Formally, this is a question about the value of an annuity. An annuity is a finan- 
cial instrument that pays out a fixed amount of money at the beginning of every year 
for some specified number of years. In particular, an n-year, m-payment annuity 
pays m dollars at the start of each year for n years. In some cases, n is finite, but 
not always. Examples include lottery payouts, student loans, and home mortgages. 
There are even Wall Street people who specialize in trading annuities. 1 

A key question is, "What is an annuity worth?" For example, lotteries often pay 
out jackpots over many years. Intuitively, $50,000 a year for 20 years ought to be 
worth less than a million dollars right now. If you had all the cash right away, you 
could invest it and begin collecting interest. But what if the choice were between 
$50,000 a year for 20 years and a half million dollars today? Now it is not clear 
which option is better. 

9.1.1 The Future Value of Money 

In order to answer such questions, we need to know what a dollar paid out in the 
future is worth today. To model this, let's assume that money can be invested at a 
fixed annual interest rate p. We'll assume an 8% rate 2 for the rest of the discussion. 

Here is why the interest rate p matters. Ten dollars invested today at interest rate 
p will become (1 + p) ■ 10 = 10.80 dollars in a year, (1 + p) 2 ■ 10 ss 1 1.66 dollars 
in two years, and so forth. Looked at another way, ten dollars paid out a year from 
now is only really worth 1/(1 + /?) - 10 ^ 9.26 dollars today. The reason is that if 
we had the $9.26 today, we could invest it and would have $10.00 in a year anyway. 
Therefore, p determines the value of money paid out in the future. 

So for an «-year, m-payment annuity, the first payment of m dollars is truly worth 
m dollars. But the second payment a year later is worth only m/(l + p) dollars. 
Similarly, the third payment is worth m/(l + p) 2 , and the n-th payment is worth 
only m/(l + p) n ~ l . The total value, V, of the annuity is equal to the sum of the 

^uch trading ultimately led to the subprime mortgage disaster in 2008-2009. We'll talk more 
about that in Section 19.5.3. 

2 U.S. interest rates have dropped steadily for several years, and ordinary bank deposits now earn 
around 1.5%. But just a few years ago the rate was 8%; this rate makes some of our examples a little 
more dramatic. The rate has been as high as 17% in the past thirty years. 
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9.1. The Value of an Annuity 

payment values. This gives: 



m 



i-1 



"- 1 / { \j 

= m ■ ^2 — J (substitute j = i — 1) 

n-l 

= m ■ ^ x j (substitute x = 1/(1 + p)). (9.2) 

7=0 

The goal of the preceding substitutions was to get the summation into a simple 
special form so that we can solve it with a general formula. In particular, the terms 
of the sum 

n-l 

^2 x J = 1 + x + x 2 + x 3 H h x n ~ l 

j=o 

form a geometric series, which means that the ratio of consecutive terms is always 
the same and it is a positive value less than one. In this case, the ratio is always x, 
and < x < 1 since we assumed that p > 0. It turns out that there is a nice 
closed-form expression for any geometric series; namely 

£y = LJL (9.3) 

Equation 9.3 can be verified by induction, but, as is often the case, the proof by 
induction gives no hint about how the formula was found in the first place. So we'll 
take this opportunity to describe a method that you could use to figure it out for 
yourself. It is called the Perturbation Method. 

9.1.2 The Perturbation Method 

Given a sum that has a nice structure, it is often useful to "perturb" the sum so that 
we can somehow combine the sum with the perturbation to get something much 
simpler. For example, suppose 

S = l + x + x 2 + --- + X"- 1 . 

An example of a perturbation would be 

xS = x + x 2 H V x n . 
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The difference between S and xS is not so great, and so if we were to subtract x S 
from S, there would be massive cancellation: 

S = \ + x + x 2 + x 3 -\ + x n ~ l 

xS — X X X * X X . 

The result of the subtraction is 

S-xS = 1 -x n . 
Solving for S gives the desired closed-form expression in Equation 9.3: 

1 - x" 
1 — X 

We'll see more examples of this method when we introduce generating functions 
in Chapter 12. 

9.1.3 A Closed Form for the Annuity Value 

Using Equation 9.3, we can derive a simple formula for V, the value of an annuity 
that pays m dollars at the start of each year for n years. 

(by Equations 9.2 and 9.3) (9.4) 

(substituting x = 1/(1 + p)). (9.5) 

Equation 9.5 is much easier to use than a summation with dozens of terms. For 
example, what is the real value of a winning lottery ticket that pays $50,000 per 
year for 20 years? Plugging in m = $50,000, n = 20, and p = 0.08 gives 
V $530,180. So because payments are deferred, the million dollar lottery is 
really only worth about a half million dollars! This is a good trick for the lottery 
advertisers. 

9.1.4 Infinite Geometric Series 

The question we began with was whether you would prefer a million dollars today 
or $50,000 a year for the rest of your life. Of course, this depends on how long 
you live, so optimistically assume that the second option is to receive $50,000 a 
year forever. This sounds like infinite money! But we can compute the value of an 
annuity with an infinite number of payments by taking the limit of our geometric 
sum in Equation 9.3 as n tends to infinity. 



— (^) 
= h + p-^/^ + p)) n ~ l \ 
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Theorem 9.1.1. If\x\ < 1, then 



i=0 



1 -x 



Proof. 



n-l 

i 



Ex ::= lim > x 
n^oo ^ 
i=0 1=0 

1 - x n 

= lim (by Equation 9.3) 

n^oo 1 — x 
1 



1-X 

The final line follows from that fact that lim„^oo x n = when \x\ < 1. ■ 

In our annuity problem, x = 1/(1 + p) < 1, so Theorem 9.1.1 applies, and we 
get 

oo 

V = m • ^ (by Equation 9.2) 
;=o 
1 

= m • (by Theorem 9.1.1) 

1 — x 

= m-i±Z (x = 1/(1 + /?)). 

Plugging in m = $50,000 and p = 0.08, we see that the value V is only $675,000. 
Amazingly, a million dollars today is worth much more than $50,000 paid every 
year forever! Then again, if we had a million dollars today in the bank earning 8% 
interest, we could take out and spend $80,000 a year forever. So on second thought, 
this answer really isn't so amazing. 

9.1.5 Examples 

Equation 9.3 and Theorem 9.1.1 are incredibly useful in computer science. In fact, 
we already used Equation 9.3 implicitly when we claimed in Chapter 6 than an 
Af-input complete binary tree has 

l + 2 + 4 + -- - + JV = 2jV-l 
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nodes. Here are some other common sums that can be put into closed form using 
Equation 9.3 and Theorem 9.1.1: 



/lY 1 
1 + 1/2+1/4 + ..._W-) =^ r 2 



(9.6) 



0.99999- •• 




1 = 



If the terms in a geometric sum grow smaller, as in Equation 9.6, then the sum is 
said to be geometrically decreasing. If the terms in a geometric sum grow progres- 
sively larger, as in Equations 9.9 and 9.10, then the sum is said to be geometrically 
increasing. In either case, the sum is usually approximately equal to the term in the 
sum with the greatest absolute value. For example, in Equations 9.6 and 9.8, the 
largest term is equal to 1 and the sums are 2 and 2/3, both relatively close to 1. In 
Equation 9.9, the sum is about twice the largest term. In Equation 9.10, the largest 
term is 3" _1 and the sum is (3 n — l)/2, which is only about a factor of 1.5 greater. 
You can see why this rule of thumb works by looking carefully at Equation 9.3 and 
Theorem 9.1.1. 



9.1.6 Variations of Geometric Sums 

We now know all about geometric sums — if you have one, life is easy. But in 
practice one often encounters sums that cannot be transformed by simple variable 
substitutions to the form J] x l . 

A non-obvious, but useful way to obtain new summation formulas from old is 
by differentiating or integrating with respect to x. As an example, consider the 
following sum: 

n-1 

^2 ix l = x + 2x 2 + 3x 3 -\ h (n - \)x n ~ l 

i = \ 
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This is not a geometric sum, since the ratio between successive terms is not fixed, 
and so our formula for the sum of a geometric sum cannot be directly applied. But 
suppose that we differentiate Equation 9.3: 



d 

dx 



\i=0 

The left-hand side of Equation 9.1 1 is simply 

n—\ , n—l 

E^'> = !>'-'• 

1=0 1=0 

The right-hand side of Equation 9. 1 1 is 

-nx n ~ x {\ - x) - (-1)(1 - x") __ -nx"' 1 + nx n + 1 - x" 
(1 -x) 2 ~ (1 -x) 2 

\-nx n ~ x + (n-l)x n 

~ (T^xJ 2 • 

Hence, Equation 9.1 1 means that 

" ' _ 1 - nx n ~ l + (n - l)x n 



E 



IX 



(1-x) 2 

1=0 v ' 

Often, differentiating or integrating messes up the exponent of x in every term. 
In this case, we now have a formula for a sum of the form J] ix'~ l , but we want a 
formula for the series J] ix'. The solution is simple: multiply by x. This gives: 

g.^-x. + C !).■+■ 

and we have the desired closed-form expression for our sum 3 . It's a little compli- 
cated looking, but it's easier to work with than the sum. 

Notice that if |x| < 1, then this series converges to a finite value even if there are 
infinitely many terms. Taking the limit of Equation 9.12 as n tends infinity gives 
the following theorem: 



3 Since we could easily have made a mistake in the calculation, it is always a good idea to go back 
and validate a formula obtained this way with a proof by induction. 
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Theorem 9.1.2. If\x\ < 1, then 



oo 

; = 1 v ' 



As a consequence, suppose that there is an annuity that pays im dollars at the 
end of each year i forever. For example, if m = $50,000, then the payouts are 
$50,000 and then $100,000 and then $150,000 and so on. It is hard to believe that 
the value of this annuity is finite! But we can use Theorem 9.1.2 to compute the 
value: 



im 



= m 



= m 



fri (i + p) 1 

!/(! + />) 



p 2 ■ 



The second line follows by an application of Theorem 9.1.2. The third line is 
obtained by multiplying the numerator and denominator by (1 + p) 2 . 

For example, if m = $50,000, and p = 0.08 as usual, then the value of the 
annuity is V = $8,437,500. Even though the payments increase every year, the in- 
crease is only additive with time; by contrast, dollars paid out in the future decrease 
in value exponentially with time. The geometric decrease swamps out the additive 
increase. Payments in the distant future are almost worthless, so the value of the 
annuity is finite. 

The important thing to remember is the trick of taking the derivative (or integral) 
of a summation formula. Of course, this technique requires one to compute nasty 
derivatives correctly, but this is at least theoretically possible! 



9.2 Power Sums 

In Chapter 3, we verified the formula 



± i = '^±A. (9.13) 



1 = 1 



But the source of this formula is still a mystery. Sure, we can prove it is true using 
well ordering or induction, but where did the expression on the right come from in 
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the first place? Even more inexplicable is the closed form expression for the sum 
of consecutive squares: 

+ do. + » (914) 

;' = 1 

It turns out that there is a way to derive these expressions, but before we explain 
it, we thought it would be fun 4 to show you how Gauss proved Equation 9. 13 when 
he was a young boy. 5 

Gauss's idea is related to the perturbation method we used in Section 9.1.2. Let 

n 
i = l 

Then we can write the sum in two orders: 

5 = 1+ 2 +...+(«- 1) + n, 
S = n + («-l) + ...+ 2 +1. 

Adding these two equations gives 

2S = (n + 1) + (n + 1) + • • • + (n + 1) + (n + 1) 
= n(n + 1). 

Hence, 

_ n(n + 1) 



Not bad for a young child. Looks like Gauss had some potential. . . 

Unfortunately, the same trick does not work for summing consecutive squares. 
However, we can observe that the result might be a third-degree polynomial in n, 
since the sum contains n terms that average out to a value that grows quadratically 
in n. So we might guess that 

n 

i 2 = an 3 + bn 2 + cn + d. 

i = l 

If the guess is correct, then we can determine the parameters a, b, c, and d by 
plugging in a few values for n. Each such value gives a linear equation in a, b, 

4 Remember that we are mathematicians, so our definition of "fun" may be different than yours. 
5 We suspect that Gauss was probably not an ordinary boy. 
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implies 
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= 1 


implies 


n 


= 2 


implies 


n 


= 3 


implies 
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c, and d. If we plug in enough values, we may get a linear system with a unique 
solution. Applying this method to our example gives: 



1 =a+b+c+d 
5 = 8a + 4b + 2c + d 
14 = 27a + 9b + 3c + d. 

Solving this system gives the solution a = 1/3, b = 1/2, c = 1/6, d = 0. 
Therefore, if our initial guess at the form of the solution was correct, then the 
summation is equal to n 3 /3 + n 2 /2 + n/6, which matches Equation 9.14. 

The point is that if the desired formula turns out to be a polynomial, then once 
you get an estimate of the degree of the polynomial, all the coefficients of the 
polynomial can be found automatically. 

Be careful! This method let's you discover formulas, but it doesn't guarantee 
they are right! After obtaining a formula by this method, it's important to go back 
and prove it using induction or some other method, because if the initial guess at 
the solution was not of the right form, then the resulting formula will be completely 
wrong! 6 



9.3 Approximating Sums 

Unfortunately, it is not always possible to find a closed-form expression for a sum. 
For example, consider the sum 

n 
i = l 

No closed form expression is known for S. 

In such cases, we need to resort to approximations for S if we want to have a 
closed form. The good news is that there is a general method to find closed-form 
upper and lower bounds that work for most any sum. Even better, the method is 
simple and easy to remember. It works by replacing the sum by an integral and 
then adding either the first or last term in the sum. 



6 Alternatively, you can use the method based on generating functions described in Chapter 12, 
which does not require any guessing at all. 
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Theorem 9.3.1. Let f : R + -> R + be a nondecreasing 1 continuous function and 
let 

n 

s = £/(0 

and 




Then 

I + /(l) < S < I + /(h). 
Similarly, if f is nonincreasing, then 

I + /(») < S < I + /(l). 

Proo/ Let / : R + -> R + be a nondecreasing function. For example, / (x) = 
is such a function. 

Consider the graph shown in Figure 9.1. The value of 

n 

s = £/(0 

is represented by the shaded area in this figure. This is because the z'th rectangle in 
the figure (counting from left to right) has width 1 and height f(i). 
The value of 

I = J f(x)dx 

is the shaded area under the curve of / (x) from 1 to n shown in Figure 9.2. 

Comparing the shaded regions in Figures 9.1 and 9.2, we see that S is at least 
/ plus the area of the leftmost rectangle. Hence, 

S > / + f{\) (9.15) 

This is the lower bound for S. We next derive the upper bound. 

Figure 9.3 shows the curve of / (x) from 1 to n shifted left by 1. This is the same 
as the curve fix + 1) from to n — 1 and it has the same area /. 

Comparing the shaded regions in Figures 9.1 and 9.3, we see that S is at most 
/ plus the area of the rightmost rectangle. Hence, 

S < I + f(n). (9.16) 

7 A function / is nondecreasing if f{x) > f(y) whenever x > y. It is nonincreasing if /(x) < 
fiy) whenever x > y. 
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1 2 3 n-2 n-l 



Figure 9.2 The shaded area under the curve of / (x) from 1 to n (shown in bold) 
is / = f" f(x) dx. 
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1 2 3 n-2 n-l 



Figure 9.3 The shaded area under the curve of f(x + 1) from to n — 1 is /, the 
same as the area under the curve of / (x) from 1 to n. This curve is the same as the 
curve in Figure 9.2 except that has been shifted left by 1. 



Combining Equations 9.15 and 9.16, we find that 

/ + /(I) < S < I + f(n), 

for any nondecreasing function /, as claimed 

The argument for the case when /is nonincreasing is very similar. The analo- 
gous graphs to those shown in Figures 9.1-9.3 are provided in Figure 9.4. As you 
can see by comparing the shaded regions in Figures 9.4(a) and 9.4(b), 

s < / + /(i). 

Similarly, comparing the shaded regions in Figures 9.4(a) and 9.4(c) reveals that 

S > I + fin). 

Hence, if / is nonincreasing, 

/ + An) <S<I + f{\). 

as claimed. ■ 

Theorem 9.3.1 provides good bounds for most sums. At worst, the bounds will 
be off by the largest term in the sum. For example, we can use Theorem 9.3.1 to 
bound the sum 

n 
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9.4. Hanging Out Over the Edge 

as follows. 

We begin by computing 



x 



dx 

3/2 



3/2 



We then apply Theorem 9.3.1 to conclude that 

^(n 3/2 -l) + l < S < ^(n 3/2 -l) + ^ 



and thus that 



\n" 2 + l - < S < ?„3/2 

3 3 3 3 



In other words, the sum is very close to |« 3 ^ 2 - 

We'll be using Theorem 9.3.1 extensively going forward. At the end of this 
chapter, we will also introduce some notation that expresses phrases like "the sum 
is very close to" in a more precise mathematical manner. But first, we'll see how 
Theorem 9.3.1 can be used to resolve a classic paradox in structural engineering. 



9.4 Hanging Out Over the Edge 

Suppose that you have n identical blocks 8 and that you stack them one on top of 
the next on a table as shown in Figure 9.5. Is there some value of n for which it is 
possible to arrange the stack so that one of the blocks hangs out completely over 
the edge of the table without having the stack fall over? (You are not allowed to use 
glue or otherwise hold the stack in position.) 

Most people's first response to this question — sometimes also their second and 
third responses — is "No. No block will ever get completely past the edge of the 
table." But in fact, if n is large enough, you can get the top block to stick out as far 
as you want: one block-length, two block- lengths, any number of block- lengths ! 

8 We will assume that the blocks are rectangular, uniformly weighted and of length 1. 
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Figure 9.5 A stack of 5 identical blocks on a table. The top block is hanging out 
over the edge of the table, but if you try stacking the blocks this way, the stack will 
fall over. 

9.4.1 Stability 

A stack of blocks is said to be stable if it will not fall over of its own accord. For 
example, the stack illustrated in Figure 9.5 is not stable because the top block is 
sure to fall over. This is because the center or mass of the top block is hanging out 
over air. 

In general, a stack of n blocks will be stable if and only if the center of mass of 
the top i blocks sits over the (i + l)st block for i = 1, 2, . . . , n — 1, and over the 
table for i = n. 

We define the overhang of a stable stack to be the distance between the edge of 
the table and the rightmost end of the rightmost block in the stack. Our goal is thus 
to maximize the overhang of a stable stack. 

For example, the maximum possible overhang for a single block is 1/2. That is 
because the center of mass of a single block is in the middle of the block (which is 
distance 1/2 from the right edge of the block). If we were to place the block so that 
its right edge is more than 1/2 from the edge of the table, the center of mass would 
be over air and the block would tip over. But we can place the block so the center 
of mass is at the edge of the table, thereby achieving overhang 1/2. This position 
is illustrated in Figure 9.6. 
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9.4. Hanging Out Over the Edge 

center of mass 




Figure 9.6 One block can overhang half a block length. 

In general, the overhang of a stack of blocks is maximized by sliding the entire 
stack rightward until its center of mass is at the edge of the table. The overhang 
will then be equal to the distance between the center of mass of the stack and the 
rightmost edge of the rightmost block. We call this distance the spread of the stack. 
Note that the spread does not depend on the location of the stack on the table — it 
is purely a property of the blocks in the stack. Of course, as we just observed, 
the maximum possible overhang is equal to the maximum possible spread. This 
relationship is illustrated in Figure 9.7. 

9.4.2 A Recursive Solution 

Our goal is to find a formula for the maximum possible spread S n that is achievable 
with a stable stack of n blocks. 

We already know that S\ = 1/2 since the right edge of a single block with 
length 1 is always distance 1/2 from its center of mass. Let's see if we can use a 
recursive approach to determine S„ for all n. This means that we need to find a 
formula for S n in terms of S; where i < n. 

Suppose we have a stable stack S of n blocks with maximum possible spread S n . 
There are two cases to consider depending on where the rightmost block is in the 
stack. 
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center of mass 




Figure 9.7 The overhang is maximized by maximizing the spread and then plac- 
ing the stack so that the center of mass is at the edge of the table. 
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Case 1: The rightmost block in S is the bottom block. Since the center of mass 
of the top n — 1 blocks must be over the bottom block for stability, the spread is 
maximized by having the center of mass of the top n — 1 blocks be directly over the 
left edge of the bottom block. In this case the center of mass of S is 9 



(n - 1) • 1 + (1) • \ , \_ 

2n 



2 =1 



to the left of the right edge of the bottom block and so the spread for S is 

1 - ^- (9-17) 
In 

For example, see Figure 9.8. 

In fact, the scenario just described is easily achieved by arranging the blocks as 
shown in Figure 9.9, in which case we have the spread given by Equation 9.17. For 
example, the spread is 3/4 for 2 blocks, 5/6 for 3 blocks, 7/8 for 4 blocks, etc. 

Can we do any better? The best spread in Case 1 is always less than 1, which 
means that we cannot get a block fully out over the edge of the table in this scenario. 
Maybe our intuition was right that we can't do better. Before we jump to any false 
conclusions, however, let's see what happens in the other case. 

Case 2: The rightmost block in S is among the top n — 1 blocks. In this case, the 
spread is maximized by placing the top n — 1 blocks so that their center of mass is 
directly over the right end of the bottom block. This means that the center of mass 
for S is at location 

{n -\).c + \.{c-\) =c 1 

n 2n 

where C is the location of the center of mass of the top n — 1 blocks. In other 
words, the center of mass of S is l/2« to the left of the center of mass of the top 
n — 1 blocks. (The difference is due to the effect of the bottom block, whose center 
of mass is 1/2 unit to the left of C.) This means that the spread of S is l/2n 
greater than the spread of the top n — 1 blocks (because we are in the case where 
the rightmost block is among the top n — 1 blocks.) 

Since the rightmost block is among the top n — 1 blocks, the spread for S is 
maximized by maximizing the spread for the top n — 1 blocks. Hence the maximum 
spread for S in this case is 

Sn-x + ^ (9-18) 
2n 



9 The center of mass of a stack of blocks is the average of the centers of mass of the individual 
blocks. 
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top n—l 
blocks 



bottom block 



5H 

Figure 9.8 The scenario where the bottom block is the rightmost block. In this 
case, the spread is maximized by having the center of mass of the top n — l blocks 
be directly over the left edge of the bottom block. 
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n-l 
blocks 



Figure 9.9 A method for achieving spread (and hence overhang) 1 — 1 /2n with 
n blocks, where the bottom block is the rightmost block. 
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where S n -\ is the maximum possible spread for n — 1 blocks (using any strategy). 

We are now almost done. There are only two cases to consider when designing 
a stack with maximum spread and we have analyzed both of them. This means 
that we can combine Equation 9.17 from Case 1 with Equation 9.18 from Case 2 to 
conclude that 

S n =max jl- S n -i + -U (9.19) 



2n 2n ' 

for any n > 1 . 

Uh-oh. This looks complicated. Maybe we are not almost done after all! 

Equation 9.19 is an example of a recurrence. We will describe numerous tech- 
niques for solving recurrences in Chapter 10, but, fortunately, Equation 9.19 is 
simple enough that we can solve it without waiting for all the hardware in Chap- 
ter 10. 

One of the first things to do when you have a recurrence is to get a feel for it 
by computing the first few terms. This often gives clues about a way to solve the 
recurrence, as it will in this case. 

We already know that S\ = 1/2. What about S2? From Equation 9.19, we find 
that 

(111 

S 2 = maxjl--, - + - 

= 3/4. 

Both cases give the same spread, albeit by different approaches. For example, see 
Figure 9.10. 

That was easy enough. What about S3? 

L 1 3 1 

S 3 =max 1--, - + - 



5 11 

6' 12 



= max 
11 

~ 12' 

As we can see, the method provided by Case 2 is the best. Let's check n = 4. 

S 4 = maxjl--, - + - 
25 
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1/2 3/4 




(b) 



Figure 9.10 Two ways to achieve spread (and hence overhang) 3/4 with n = 2 
blocks. The first way (a) is from Case 1 and the second (b) is from Case 2. 
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Wow! This is a breakthrough — for two reasons. First, Equation 9.20 tells us that 
by using only 4 blocks, we can make a stack so that one of the blocks is hanging 
out completely over the edge of the table. The two ways to do this are shown in 
Figure 9.11. 

The second reason that Equation 9.20 is important is that we now know that 
S4 > 1, which means that we no longer have to worry about Case 1 for n > 4 since 
Case 1 never achieves spread greater than 1 . Moreover, even for n < 4, we have 
now seen that the spread achieved by Case 1 never exceeds the spread achieved by 
Case 2, and they can be equal only for n = 1 and n = 2. This means that 

S„ = S n -i + - 1 - (9.21) 
2n 

for all n > 1 since we have shown that the best spread can always be achieved 
using Case 2. 

The recurrence in Equation 9.21 is much easier to solve than the one we started 
with in Equation 9.19. We can solve it by expanding the equation as follows: 

1 

?>n = S n -i + — 

2n 

1 1 

— S n -2 + —, 7T + — 

2(n — 1) 2n 

111 

" Sn ~ 3 + 2(J^2) + + 2n 



and so on. This suggests that 



Sn=Y,L (9-22) 



, 2i 
i = l 



which is, indeed, the case. 

Equation 9.22 can be verified by induction. The base case when n = 1 is true 
since we know that S\ = 1/2. The inductive step follows from Equation 9.21. 

So we now know the maximum possible spread and hence the maximum possible 
overhang for any stable stack of books. Are we done? Not quite. Although we 
know that S4 > 1, we still don't know how big the sum Y11=\ ji can g et - 

It turns out that S n is very close to a famous sum known as the «th Harmonic 
number H n . 

9.4.3 Harmonic Numbers 



Definition 9.4.1. The nth Harmonic number is 

n 



i = l 
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1/2 




1/8 1/6 1/4 1/2 



LJ 

(b) 

Figure 9.11 The two ways to achieve spread (and overhang) 25/24. The method 
in (a) uses Case 1 for the top 2 blocks and Case 2 for the others. The method in (b) 
uses Case 2 for every block that is added to the stack. 
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So Equation 9.22 means that 



S n = *y- (9-23) 



The first few Harmonic numbers are easy to compute. For example, 

1 1 1 25 

H 4 = I + - + - + - = —. 

2 3 4 12 

There is good news and bad news about Harmonic numbers. The bad news is that 
there is no closed-form expression known for the Harmonic numbers. The good 
news is that we can use Theorem 9.3.1 to get close upper and lower bounds on H n . 
In particular, since 



/•" 1 

/ — dx = ln(x) 
Ji x 



l 



Theorem 9.3.1 means that 



Hn) + ~<H n < ln(n) + 1. (9.24) 
n 

In other words, the «th Harmonic number is very close to ln(n). 

Because the Harmonic numbers frequently arise in practice, mathematicians 
have worked hard to get even better approximations for them. In fact, it is now 
known that 

H „ = ln( „ ) + 7+ _L + _i_ + _lW (9 . 2 5) 

Here y is a value 0.577215664 . . . called Euler's constant, and e{n) is between 
and 1 for all n. We will not prove this formula. 

We are now finally done with our analysis of the block stacking problem. Plug- 
ging the value of H n into Equation 9.23, we find that the maximum overhang for 
n blocks is very close to | ln(«). Since ln(n) grows to infinity as n increases, this 
means that if we are given enough blocks (in theory anyway), we can get a block to 
hang out arbitrarily far over the edge of the table. Of course, the number of blocks 
we need will grow as an exponential function of the overhang, so it will probably 
take you a long time to achieve an overhang of 2 or 3, never mind an overhang 
of 100. 

9.4.4 Asymptotic Equality 

For cases like Equation 9.25 where we understand the growth of a function like H n 
up to some (unimportant) error terms, we use a special notation, ~, to denote the 
leading term of the function. For example, we say that H„ ~ ln(n) to indicate that 
the leading term of H n is ln(n). More precisely: 
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Definition 9.4.2. For functions /, g : R R, we say / is asymptotically equal to 
g, in symbols, 

fix) ~ g(x) 



iff 



lim /(*)/gO) = 1. 



Although it is tempting to write H„ ~ ln(«) + y to indicate the two leading 
terms, this is not really right. According to Definition 9.4.2, H n ~ ln(n) + c where 
c is any constant. The correct way to indicate that y is the second-largest term is 
H n - ln(n) ~ y. 

The reason that the ~ notation is useful is that often we do not care about lower 
order terms. For example, if n = 100, then we can compute H(n) to great precision 
using only the two leading terms: 



\H n -ln(w) - y\ 



< 



1 1 1 

+ 



200 120000 120 • 100 4 



1 

< 200' 



We will spend a lot more time talking about asymptotic notation at the end of the 
chapter. But for now, let's get back to sums. 



9.5 Double Trouble 

Sometimes we have to evaluate sums of sums, otherwise known as double summa- 
tions. This sounds hairy, and sometimes it is. But usually, it is straightforward — 
you just evaluate the inner sum, replace it with a closed form, and then evaluate the 
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outer sum (which no longer has a summation inside it). For example, 



10 



n=0 \ i=0 



ei» , E"')-e(^) 

Kt^)E/-(t^)E/*" + ' 



Equation 9.3 





1 






(1 - 


x)(l- 

1 


y) 


(if 


(1 - 


x)(l- 


y) 




(1 


-xy)- 


-x(l 


-J) 


(1 - 


x)(l- 
1 - 


J)(l 

- X 


-xj) 


(1 - 


x)(l- 

1 


J)(l 


-xj) 


(1 - 


J)(l- 


xj)' 





«=0 

oo 

n=0 



Theorem 9.1.1 



Theorem 9.1.1 



When there's no obvious closed form for the inner sum, a special trick that is 
often useful is to try exchanging the order of summation. For example, suppose we 
want to compute the sum of the first n Harmonic numbers 



n k 



E**-EEy 

k=lj=l J 



(9.26) 



k=i 



For intuition about this sum, we can apply Theorem 9.3.1 to Equation 9.24 to con- 
clude that the sum is close to 



ln(x) dx = x ln(x) — x 



= n ln(n) — n + 1. 



Now let's look for an exact answer. If we think about the pairs (k, j) over which 



10 Ok, so maybe this one is a little hairy, but it is also fairly straightforward. Wait till you see the 
next one! 
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we are summing, they form a triangle: 





j 
1 


2 


3 


4 5.. 


n 


k 1 


1 










2 


1 


1/2 








3 


1 


1/2 


1/3 






4 


1 


1/2 


1/3 


1/4 




n 


1 


1/2 






l/n 



The summation in Equation 9.26 is summing each row and then adding the row 
sums. Instead, we can sum the columns and then add the column sums. Inspecting 
the table we see that this double sum can be written as 



n k 



k=l 



k=ij=i J 

n n . 



j=ik=j J 
n 1 n 

= E}E' 

7=1 J k=J 
7=1 ' 

n 

= E 



J 

n + 1 



7=1 



J 



^7 

7=1 J 



n 1 n 

= (" + 1 )Ey-E 1 

7=1 J 7=1 
= (n + \)H n -n. 



(9.27) 
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9.6 Products 




We've covered several techniques for rinding closed forms for sums but no methods 
for dealing with products. Fortunately, we do not need to develop an entirely new 
set of tools when we encounter a product such as 


n\::=f\i. 

i = l 


(9.28) 


That's because we can convert any product into a sum by taking a logarithm. For 
example, if 

n 


p = U m, 

i = \ 




then 

n 




ln(P) = £>(/(/)). 

7 = 1 




We can then apply our summing tools to find a closed form (or approximate closed 
form) for ln(P) and then exponentiate at the end to undo the logarithm. 

For example, let's see how this works for the factorial function n\ We start by 
taking the logarithm: 


ln(n!) = ln(l-2-3---(«-l)-n) 

= ln(l) + ln(2) + ln(3) + • • • + ln(n - 1) + ln(«) 


n 

= £>(/). 




Unfortunately, no closed form for this sum is known. However, we can apply 
Theorem 9.3.1 to find good closed-form bounds on the sum. To do this, we first 
compute 


r 

/ ln(x) dx = x ln(x) — x 


n 
1 


= n ln(«) — n + 1. 


Plugging into Theorem 9.3.1, this means that 




n 

n\n(n) — n + l < ^ln(z') < n\n(n) 

i = \ 


-n + l +ln(n). 
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Exponentiating then gives 



n n n n+i 

T < n < r . (9.29) 

e n-l ~ ~ e n-\ 

n I „n— 1 



This means that n! is within a factor of n of n n /e 
9.6.1 Stirling's Formula 

n! is probably the most commonly used product in discrete mathematics, and so 
mathematicians have put in the effort to find much better closed-form bounds on its 
value. The most useful bounds are given in Theorem 9.6.1. 



Theorem 9.6.1 (Stirling's Formula). For all n > 1, 

where 



12n + 1 ~~ ~ Yin 

Theorem 9.6.1 can be proved by induction on n, but the details are a bit painful 
(even for us) and so we will not go through them here. 

There are several important things to notice about Stirling's Formula. First, e(n) 
is always positive. This means that 

n\>V2nn(^y (9.30) 

for all n e N + . 

Second, e(n) tends to zero as n gets large. This means that 11 

n\~V2nn(^y, (9.31) 

which is rather surprising. After all, who would expect both n and e to show up in 
a closed-form expression that is asymptotically equal to «!? 

Third, e(n) is small even for small values of n. This means that Stirling's For- 
mula provides good approximations for n! for most all values of n. For example, if 
we use 



The ~ notation was defined in Section 9.4.4. 
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Approximation n > 1 « > 10 n > 100 n > 1000 

<Jlnn{^) n < 10% < 1% <0.1% <0.01% 

V2xn (j)" e l/l2n < 1% <0.01% < 0.0001% < 0.000001% 

Table 9.1 Error bounds on common approximations for n \ from Theorem 9.6.1. 
For example, if n > 100, then \pht~n (|) approximates n \ to within 0.1%. 



as the approximation for n\, as many people do, we are guaranteed to be within a 
factor of 



of the correct value. For n > 10, this means we will be within 1% of the correct 
value. For n > 100, the error will be less than 0.1%. 

If we need an even closer approximation for n!, then we could use either 



'n\ n 

or 



V2^(*)V/ 12 " 



'«V„1/(12» + 1) 

depending on whether we want an upper bound or a lower bound, respectively. By 
Theorem 9.6.1, we know that both bounds will be within a factor of 

_J !_ > 

g 12n 12n + J — gl44« 2 + 12« 

of the correct value. For n > 10, this means that either bound will be within 0.01% 
of the correct value. For n > 100, the error will be less than 0.0001%. 

For quick future reference, these facts are summarized in Corollary 9.6.2 and 
Table 9.1. 

Corollary 9.6.2. For n > 1, 



Forn > 10, 
Forn > 100, 



n\ < 1.009V2;rn 
n\ < 1.0009V^jr« (-) 
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9.7 Asymptotic Notation 

Asymptotic notation is a shorthand used to give a quick measure of the behavior of 
a function / (n) as n grows large. For example, the asymptotic notation ~ of Defi- 
nition 9.4.2 is a binary relation indicating that two functions grow at the same rate. 
There is also a binary relation indicating that one function grows at a significantly 
slower rate than another. 

9.7.1 Little Oh 

Definition 9.7.1. For functions /, g : R -> R, with g nonnegative, we say / is 
asymptotically smaller than g, in symbols, 

f(x) = o(g(x)), 

iff 

lim f(x)/g(x) = 0. 

For example, 1000.x 1 ' 9 = o(x 2 ), because 1000x 1 9 /x 2 = 1000/x 1 and since 
x 01 goes to infinity with x and 1000 is constant, we have lim x ^oo 1000x 1 9 /x 2 = 
0. This argument generalizes directly to yield 

Lemma 9.7.2. x a = o(x b ) for all nonnegative constants a < b. 

Using the familiar fact that log x < x for all x > 1 , we can prove 

Lemma 9.7.3. logx = o(x € ) for all e > 0. 

Proof. Choose e > 8 > and let x = z 8 in the inequality log x < x. This implies 
logz < z s /8 = o(z e ) by Lemma 9.7.2. (9.32) 



Corollary 9.7.4. x b = o{a x ) for any a,b el with a > 1. 

Lemma 9.7.3 and Corollary 9.7.4 can also be proved using l'Hopital's Rule or 
the McLaurin Series for log x and e x . Proofs can be found in most calculus texts. 
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9.7.2 Big Oh 

Big Oh is the most frequently used asymptotic notation. It is used to give an upper 
bound on the growth of a function, such as the running time of an algorithm. 

Definition 9.7.5. Given nonnegative functions /, g : R R, we say that 

/ = 0(g) 

iff 

limsup f(x)/g(x) < oo. 

This definition 12 makes it clear that 
Lemma 9.7.6. If f = o(g) or f ~ g, then f = 0(g). 

Proof, lim f/g = or lim f/g = 1 implies lim f/g < oo. ■ 

It is easy to see that the converse of Lemma 9.7.6 is not true. For example, 
2x = O(x), but 2x / x and 2x ^ o(x). 

The usual formulation of Big Oh spells out the definition of lim sup without 
mentioning it. Namely, here is an equivalent definition: 

Definition 9.7.7. Given functions /, g : R -> R, we say that 

/ = 0(g) 

iff there exists a constant c > and an xo such that for all x > xq, \f(x)\ < cg(x). 

This definition is rather complicated, but the idea is simple: f(x) = 0(g(x)) 
means f(x) is less than or equal to g(x), except that we're willing to ignore a 
constant factor, namely, c, and to allow exceptions for small x, namely, x < Xq. 

We observe, 

Lemma 9.7.8. If f = o(g), then it is not true that g = 0(f ). 

12 We can't simply use the limit as x ->• oo in the definition of 0(), because if / (x)/g(x) oscillates 
between, say, 3 and 5 as x grows, then / = 0(g) because / < 5g, but lim^^oo f(x)/g(x) 
does not exist. So instead of limit, we use the technical notion of lim sup. In this oscillating case, 
limsup^^^ f(x)/g(x) = 5. 

The precise definition of lim sup is 

limsupA(x) ::= lim \ub y > x h(y), 
where "lub" abbreviates "least upper bound." 
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Proof. 

]im ?W = 1 = I = oo 

x^oo f( x ) lim x ^oo f(x)/g(x) 

sog^ 0(f). 

m 

Proposition 9.7.9. 100x 2 = 0(x 2 ). 

Proof. Choose c = 100 and xq = 1. Then the proposition holds, since for all 
x > 1, |l00x 2 | < lOOx 2 . ■ 

Proposition 9.7.10. x 2 + lOOx + 10 = 0(x 2 ). 

Proof. (x 2 + 100x + 10)/x 2 = 1 + 100/x + 10/x 2 and so its limit as x approaches 
infinity is 1+0+0 = 1. So in fact, x 2 + 100% + 10 ~ x 2 , and therefore x 2 + 100x + 
10 = 0(x 2 ). Indeed, it's conversely true that x 2 = 0(x 2 + lOOx + 10). ■ 

Proposition 9.7.10 generalizes to an arbitrary polynomial: 

Proposition 9.7.11. a^x k + a^- X x k ~ x + • • • + a\x + «o = 0(x k ). 

We'll omit the routine proof. 

Big Oh notation is especially useful when describing the running time of an 
algorithm. For example, the usual algorithm for multiplying n x n matrices uses 
a number of operations proportional to n 3 in the worst case. This fact can be 
expressed concisely by saying that the running time is 0(n 3 ). So this asymptotic 
notation allows the speed of the algorithm to be discussed without reference to 
constant factors or lower-order terms that might be machine specific. It turns out 
that there is another, ingenious matrix multiplication procedure that uses 0(n 2 - 55 ) 
operations. This procedure will therefore be much more efficient on large enough 
matrices. Unfortunately, the (9(« 2 55 )-operation multiplication procedure is almost 
never used in practice because it happens to be less efficient than the usual 0(n 3 ) 
procedure on matrices of practical size. 13 

9.7.3 Omega 

Suppose you want to make a statement of the form "the running time of the algo- 
rithm is a least. . .". Can you say it is "at least 0(n 2 )"? No! This statement is 
meaningless since big-oh can only be used for upper bounds. For lower bounds, 
we use a different symbol, called "big-Omega." 

13 It is even conceivable that there is an 0(n 2 ) matrix multiplication procedure, but none is known. 
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Definition 9.7.12. Given functions /, g : R R, we say that 

/ = «(g) 

iff there exists a constant c > and an xq such that for all x > Xo, we have 
fix) > c\g(x)\. 

In other words, f(x) = Q(g(x)) means that f(x) is greater than or equal 
to g(x), except that we are willing to ignore a constant factor and to allow ex- 
ceptions for small x. 

If all this sounds a lot like big-Oh, only in reverse, that's because big-Omega is 
the opposite of big-Oh. More precisely, 

Theorem 9.7.13. f(x) = 0(g(x)) if and only if g(x) = Q(/(x)). 
Proof. 

fix) = Oigix)) 

iff 3c > 0, xq. Vx > xq. |/(x)| < cg(x) (Definition 9.7.7) 



iff 3c> 0,x . Vx > xq. gix) > -|/0)| 

c 

iff 3c' > 0, Xq. Vx > Xq. g(x) > c'\f(x)\ 

iff g(x) = n(f(x)) 



(setc' = 1/c) 
(Definition 9.7. 12) 

^(100x + Vx). 



For example, x 2 = f2(x), 2 X = ^(x 2 ), and x/100 
So if the running time of your algorithm on inputs of size n is T(n), and you 
want to say it is at least quadratic, say 

T(n) = Q(n 2 ). 

Little Omega 

There is also a symbol called little-omega, analogous to little-oh, to denote that one 
function grows strictly faster than another function. 

Definition 9.7.14. For functions / g : R -> R with / nonnegative, we say that 

fix) = co(g(x)) 

iff 

r Si*) 

hm — — = 0. 

x^oo f( X ) 



In other words, 
iff 



fix) = co(g(x)) 
gix) = o(fix)). 
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For example, x 15 = a>(x) and */x = a>(ln 2 (x)). 

The little-omega symbol is not as widely used as the other asymptotic symbols 
we have been discussing. 

9.7.4 Theta 

Sometimes we want to specify that a running time T(n) is precisely quadratic up to 
constant factors (both upper bound and lower bound). We could do this by saying 
that T(n) = 0(n 2 ) and T(n) = £l(n 2 ), but rather than say both, mathematicians 
have devised yet another symbol, 9, to do the job. 

Definition 9.7.15. 

f = ®(g) iff / = 0(g) and g= 0(f ). 

The statement / = &(g) can be paraphrased intuitively as "/ and g are equal 
to within a constant factor." Indeed, by Theorem 9.7.13, we know that 

f = @(g) iff / = 0(g) and / = Si(g). 

The Theta notation allows us to highlight growth rates and allow suppression 
of distracting factors and low-order terms. For example, if the running time of an 
algorithm is 

T(n) = 10« 3 -20n 2 + 1, 
then we can more simply write 

T(n) = &(n 3 ). 

In this case, we would say that T is of order n 3 or that T(n) grows cubically, which 
is probably what we really want to know. Another such example is 

2 _ 7 (2.7x 113 +x 9 -86) 4 
7r 2 3 x ~ 7 + = — - 1.08 3x = &(3 X ). 

Just knowing that the running time of an algorithm is 0(n 3 ), for example, is use- 
ful, because if n doubles we can predict that the running time will by and large 14 
increase by a factor of at most 8 for large n. In this way, Theta notation preserves in- 
formation about the scalability of an algorithm or system. Scalability is, of course, 
a big issue in the design of algorithms and systems. 

14 Since 0(w 3 ) only implies that the running time, T(n), is between cn 3 and dn 3 for constants 
< c < d, the time T(2n) could regularly exceed T(n) by a factor as large as %djc. The factor is 
sure to be close to 8 for all large n only if T(n) ~ n 3 . 
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9.7.5 Pitfalls with Asymptotic Notation 

There is a long list of ways to make mistakes with asymptotic notation. This section 
presents some of the ways that Big Oh notation can lead to ruin and despair. With 
minimal effort, you can cause just as much chaos with the other symbols. 

The Exponential Fiasco 

Sometimes relationships involving Big Oh are not so obvious. For example, one 
might guess that A x = 0(2 X ) since 4 is only a constant factor larger than 2. This 
reasoning is incorrect, however; 4 X actually grows as the square of 2 X . 

Constant Confusion 

Every constant is 0(1). For example, 17 = 0(1). This is true because if we let 
/ (x) = 17 and g(x) = 1, then there exists a c > and an xo such that |/(x)| < 
cg(x). In particular, we could choose c = 17 and xq = 1, since |17| < 17-1 for all 
x > 1 . We can construct a false theorem that exploits this fact. 

False Theorem 9.7.16. 

n 
i = l 

Bogus proof. Define / (n) = Yll=i 2 = 1 + 2 + 3H h n. Since we have shown 

that every constant i is 0(1), f(n) = 0(1) + 0(1) + • • • + 0(1) = 0(n). ■ 

Of course in reality X^=i ' = n ( n + ^ 0(n). 

The error stems from confusion over what is meant in the statement i = 0(1). 
For any constant i € N it is true that i = 0(1). More precisely, if / is any constant 
function, then / = 0(1). But in this False Theorem, i is not constant — it ranges 
over a set of values 0, \,...,n that depends on n. 

And anyway, we should not be adding 0(l)'s as though they were numbers. We 
never even defined what 0(g) means by itself; it should only be used in the context 
"/ = O (g)" to describe a relation between functions / and g. 

Lower Bound Blunder 

Sometimes people incorrectly use Big Oh in the context of a lower bound. For 
example, they might say, "The running time, T(n), is at least 0(n 2 )," when they 
probably mean 15 "T(n) = Q(n 2 )" 

15 This can also be correctly expressed as n 2 = 0(T(n)), but such notation is rare. 
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9. 7. Asymptotic Notation 
Equality Blunder 

The notation / = 0(g) is too firmly entrenched to avoid, but the use of "=" 
is really regrettable. For example, if / = 0(g), it seems quite reasonable to 
write 0(g) = f. But doing so might tempt us to the following blunder: because 
2n = 0(n), we can say 0(n) = In. But n = 0(n), so we conclude that n = 
0(n) = 2n, and therefore n = In. To avoid such nonsense, we will never write 

"0(f) = g r 

Similarly, you will often see statements like 

H n = ln(ra) + y + O 

or 

n\ = (1 +o(\))yflTtn . 
In such cases, the true meaning is 

H n =\n(n) + y + f(n) 
for some f(n) where f(n) = 0(l/n),and 

n\ = (1 + g(n))V2nn 

where g(n) = o(l). These transgressions are OK as long as you (and you reader) 
know what you mean. 
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